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The  Honorable  John  Glenn  SucCtS  S 

Ranking  Minority  Member  v/ 

Committee  on  Governmental  Affairs  U  )<  c~  ) 

United  States  Senate 


Dear  Senator  Glenn: 


This  report  responds  to  your  Februaiy  1996  request  that  we  review  the 
Department  of  Energy’s  (doe)  progress  in  developing  a  new  nuclear 
materials  tracking  system.  This  new  system  is  intended  to  replace  the 
existing  Nuclear  Materials  Management  and  Safeguards  System  (nmmss), 
which  is  the  United  States’  official  system  for  tracking  U.S.  imports  and 
exports  of  nuclear  materials.  In  December  1994,  we  reported  on  doe’s 
planning  for  the  replacement  nmmss.'  This  report  discusses  (1)  what 
actions  doe  has  taken  to  implement  the  recommendations  in  our  previous 
report  and  (2)  whether  doe  is  adequately  addressing  key  system 
development  risks. 


Results  in  Brief 


DOE  has  not  implemented  any  of  the  recommendations  contained  in  our 
prior  report  and  has  no  plans  to  do  so.  In  December  1994,  we  reported  that 
DOE  had  not  adequately  planned  the  replacement  nmmss  and  recommended 
that  the  Department  determine  users’  requirements,  investigate 
alternatives,  and  conduct  cost-benefit  analyses  before  proceeding  further 
with  the  replacement  system.  However,  doe  continued  with  the  system 
development  without  performing  these  steps  because  it  believed  that  its 
planning  was  sufficient  and  that  it  would  not  be  cost-effective  to  delay  the 
replacement  system.  Due  to  its  lack  of  soimd  planning,  doe  does  not  know 
if  the  system  will  fulfill  the  needs  of  its  m^or  users  or  be  cost-effective. 

'These  planning  risks  are  magnified  by  additional  system  development 
risks  that  doe  is  not  adequately  addressing.  For  example,  the 
subcontractor  building  the  replacement  nmmss  has  not  documented  its 
system  development  process.  Because  little  system  documentation  exists, 
and  the  contract  does  not  require  any  interim  deliverables  describing 
development  progress  before  complete  system  delivery,  doe  cannot 
determine  the  status  of  the  development  effort.  In  addition,  the 
subcontractor  did  not  place  its  software  under  configuration  management. 


‘Nuclear  Nonproliferation:  U.S.  International  Nuclear  Materials  Tracking  Capabilities  Are  Limited 
(GAO/RCED/AIMD-95-5,  December  27, 1994).  ~  - - 
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Sound  configuration  management  helps  ensure  that  the  status  of  the 
system’s  software  is  known  at  all  times  and  that,  when  more  than  one 
programmer  is  making  changes  and  updating  the  software,  all  changes  are 
consistent  and  are  being  written  to  the  same  software  version.  Finally,  doe 
plans  to  pay  for,  install,  and  use  the  replacement  nmmss  without  requiring 
that  it  pass  acceptance  testing.  Acceptance  testing  demonstrates  that  a 
system  meets  hardware,  software,  and  performance  requirements  and 
users’  operational  needs.  Without  such  testing,  coupled  with  inadequate 
planning  and  the  lack  of  basic  system  development  discipline  and  sound 
practices,  doe  has  no  assurance  that  the  replacement  nmmss  will  ever 
perform  as  intended. 


Background 


NMMSS  is  the  United  States’  official  nuclear  materials  tracking  and 
accovmting  system,  nmmss  provides  information  on  nuclear  materials  to 
support  both  domestic  programs  and  international  nuclear  policies. 
Keeping  track  of  the  growing  amoimt  of  nuclear  materials  is  especially 
important  as  a  result  of  the  breakdown  of  the  Soviet  Union  and  increases 
in  both  domestic  and  international  terrorism.  Tracking  and  accounting  for 
the  hundreds  of  tons  of  plutonium,  highly  enriched  uranium,  and  other 
nuclear  materials  that  have  accumulated  are  important  to  help  (1)  ensure 
that  nuclear  materials  are  used  only  for  peaceful  pmposes,  (2)  protect 
nuclear  materials  from  loss,  theft,  or  other  diversion,  (3)  comply  with 
international  treaty  obligations,  and  (4)  provide  data  to  policymakers  and 
other  government  officials  on  the  amount  and  location  of  nuclear 
materials. 

The  NMMSS  database  contains  data  on  nuclear  materials  supplied  and 
controlled  under  international  agreements,  including  U.S.-supplied 
international  nuclear  materials  transactions,  foreign  contracts, 
import/export  licenses,  govemment-to-govemment  approvals,  and  other 
DOE  authorizations,  such  as  authorizations  to  retransfer  U.S.-supplied 
materials  between  foreign  countries,  nmmss  also  maintains  and  provides 
DOE  with  information  on  domestic  production  and  materials  management, 
safeguards,  physical  accountability,  financial  and  cost  accoimting,  and 
other  data  related  to  nuclear  materials  accountability  and  safeguards  for 
Nuclear  Regulatory  Commission  licensees. 

DOE  and  the  Nuclear  Regulatory  Commission  cosponsor  nmmss,  and  it  is 
managed  and  operated  by  a  doe  contractor — Martin  Marietta  Energy 
Systems,  Incorporated,  nmmss  has  been  used  to  account  for  U.S.  imports 
and  exports  of  nuclear  materials  since  1977. 
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Because  the  existing  nmmss  is  an  older  system,  doe  decided  to  replace  and 
modernize  it.  The  existing  nmmss  is  housed  on  a  mainframe  using 
unstructured  COBOL  code.  Performing  modifications  on  the  existing 
NMMSS  and  designing  custom  reports  are  difficult  because  of  the  volume 
and  complexity  of  the  code.  Accordingly,  doe’s  Office  of  Nonproliferation 
and  National  Security,  which  is  responsible  for  operating  nmmss,  tasked 
the  Lawrence  Livermore  National  Laboratory  with  developing  a  new 
NMMSS.  Livermore  hired  a  subcontractor  to  perform  this  task  in 
February  1994  and  assigned  a  program  manager  to  oversee  the  effort.  In 
April  1994,  Livermore  formed  a  technical  advisory  committee,  composed 
of  senior  computer  scientists  and  material  control  and  accoimtability 
specialists,  to  assist  the  program  manager  in  overseeing  the  system 
development. 

The  replacement  nmmss  is  scheduled  to  become  operational  on 
September  1, 1995.  Martin  Marietta  is  scheduled  to  discontinue  operation 
of  the  existing  nmmss  during  September  1995.  i 


Scope  and 
Methodology 


To  address  our  objectives,  we  reviewed  the  replacement  nmmss  contract, 
transition  plan,  test  plan,  and  various  other  draft  system  documents.  We 
requested  documentation  on  the  status  of  the  system  coding  and  testing; 
however,  none  was  available  for  our  review.  We  also  analyzed 
documentation  provided  to  us  by  Lawrence  Livermore’s  technical  advisory 
committee  on  the  subcontractor’s  development  efforts.  In  addition,  we 
analyzed  documentation  from  various  user  groups  on  their  concerns  with 
the  nmmss  development.  We  analyzed  doe  Order  1330.  ID,  Computer 
Software  Management,  to  determine  its  applicability  to  this  project  and 
whether  or  not  it  was  being  followed. 

We  interviewed  doe  officials  in  the  Office  of  Nonproliferation  and  National 
Security  concerning  actions  taken  to  implement  the  recommendations  in 
our  previous  report  and  the  status  of  the  nmmss  development.  We  also 
interviewed  the  nmmss  program  manager,  members  of  Lawrence 
Livermore’s  technical  advisory  committee,  contract  officials  at  doe  and 
Lawrence  Livermore,  and  the  nmmss  subcontractor’s  lead  programmers, 
system  engineer,  and  project  managers  to  determine  the  status  of  the 
system  development.  In  addition,  we  interviewed  officials  in  doe’s  Defense 
Programs  Office — ^the  biggest  user  of  nmmss  information — on  their 
concerns  about  the  replacement  nmmss  development. 
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We  performed  our  work  between  Febmaiy  1995  and  May  1995,  in 
accordance  with  generally  accepted  government  auditing  standards.  Our 
work  was  primarily  done  at  doe’s  headquarters  in  Washington,  D.C.,  and 
its  offices  in  Germantown,  Maryland;  at  Lawrence  Livermore  National 
Laboratory  in  Livermore,  Califonua;  and  at  the  subcontractor  s  facility  in 
Norcross,  Georgia.  The  Department  of  Energy  provided  written  comments 
on  a  draft  of  this  report.  These  comments  are  presented  and  evaluated  in 
the  report,  and  are  reprinted  in  appendix  1. 


DOE  Is  Pursuing  the 
Replacement  NMMSS 
Without  Addressing 
Critical  Planning 
Issues 


In  December  1994,  we  reported  that  doe  did  not  adequately  plan  the 
development  effort  for  the  replacement  nmmss.^  For  example,  doe  did  not 
follow  sound  system  development  practices  such  as  identifying  and 
defining  users’  needs  and  adequately  exploring  design  alternatives  that 
would  best  satisfy  these  needs  in  the  most  economic  fashion.  Accordingly, 
we  recommended  that  doe  determine  users’  requirements,  investigate 
alternatives,  conduct  cost-benefit  analyses,  and  develop  a  plan  to  meet 
identified  needs  before  investing  further  resources  in  the  replacement 
NMMSS. 


In  its  official  response  to  the  recommendations  in  our  prior  report,  doe 
stated  that  it  did  not  concur  with  our  recommendations  and  that  it  did  not 
believe  it  would  be  cost-effective  to  delay  its  effort  to  transition  from  the 
existing  system  to  the  new  system.  Further,  in  commenting  on  a  draft  of 
this  report,  the  Acting  Director  of  the  Office  of  Nonproliferation  and 
National  Security  stated  that  doe’s  planning  was  sufficient.  However, 
because  of  doe’s  lack  of  basic  planning,  it  does  not  know  if  the  system  will 
fulfill  the  needs  of  its  major  users  or  be  cost-effective. 


System  Development 
Risks  Are  High 


Adhering  to  generally  accepted  system  development  practices  helps  to 
ensure  that  information  systems  perform  as  desired.^  These  practices 
include  (1)  generating  clear,  complete,  and  accurate  documentation 
throughout  the  system  development  process,  (2)  placing  the  software 
development  under  configuration  management,  and  (3)  ensuring  that  the 
system  successfully  completes  acceptance  testing  prior  to  becoming 
operational.  However,  because  doe  has  not  required  the  subcontractor  to 


2GAO/RCED/AIMD-95-5,  December  27,  1994. 


^Such  practices  are  discussed  in  Defense  Acquisition  Management  Policies  and  Procedures  (DOD 
5000.2,  February  1991);  Systems  Engineering  Management  Guide  (Defense  Systems  Management 
College,  January  1990);  Capability  Maturity  Model  for  Software  (CMU/SEI-91-TR-24,  ESD-TR-91-24, 
August  1991);  Key  Practices  of  the  Capability  Maturity  ModeiTCMU/SEI-91-TR-25,  ESD-TR-91-25, 
August  1991);  and  Defense  System  Software  Development  (DQD-STD-2167A,  February  1988). 
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follow  any  of  these  practices  for  the  replacement  nmmss,  the  Department 
does  not  know  how  much  of  the  system  development  is  completed  and 
whether  the  part  that  is  completed  performs  as  required.  As  a  result,  the 
risk  of  system  failure  is  inordinately  high. 


Little  System  Development  doe  Order  1330.  id,  Computer  Software  Management,  requires  that  the 
Documentation  Exists  development  of  a  system  be  documented  so  that,  among  other  things,  the 

status  of  the  system  is  known  at  all  times.  Documentation,  such  as  the 
results  of  system  testing  and  the  tracking  of  source  code  as  it  changes, 
allows  program  managers  to  review  the  development’s  progress  and 
determine  if  requirements  are  being  met. 

The  subcontractor  developing  the  replacement  nmmss  could  not  provide 
any  system  documentation — software  specifications,  system  requirements, 
results  of  formal  reviews  (e.g.,  system/preliminary/critical  design)  or 
informal  system  testing  reports,  operational  procedures,  quality  assurance 
checklists,  or  project  tracking  reports.  Because  little  system 
documentation  exists,  and  the  contract  does  not  require  any  interim 
deliverables  that  measure  system  performance,  doe  does  not  know  the 
status  of  the  system  development.  In  addition,  members  of  Livermore’s 
technical  advisory  committee  told  us  they  have  been  unable  to  obtain  the 
documentation  they  needed  to  determine  the  status  of  the  development 
effort.  As  a  result,  the  committee  said  it  could  not  accurately  determine 
such  factors  as  the  number  of  lines  of  code  in  the  system.  In  fact,  the 
advisory  committee  could  only  estimate  system  size  in  veiy  gross 
terms — ^between  10,000  and  100,000  lines  of  code. 

DOE  officials  agreed  that  the  development  effort  is  largely  undocumented 
and  stated  that  doe  historically  has  not  enforced  its  own  regulations 
requiring  system  documentation.  At  the  conclusion  of  our  review  and  in 
conunenting  on  a  draft  of  this  report,  doe  officials  told  us  that  they  will 
begin  to  require  such  documentation  for  the  replacement  nmmss. 


Configuration  Management  a  successful  system  development  project  should  include  a  software 
Was  Not  Used  configuration  management  plan  that  clearly  defines  the  procedures  for 

identifying,  accoimting  for,  and  reporting  on  changes  to  software  items 
that  are  imder  configuration  control.  Configuration  management  is 
necessary  throughout  the  life  cycle  of  a  software  project  because  it 
provides  (1)  a  control  mechanism  to  ensure  that  the  software  status  is 
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accurately  known  at  all  times  and  (2)  a  baseline  for  system  developers  and 
testers. 

Although  the  subcontractor  developed  a  software  configuration 
management  plan  for  the  replacement  nmmss,  no  software  had  been  placed 
under  configuration  control.  As  a  result,  doe  does  not  know  what  version 
of  the  software  is  current,  which  versions  of  the  software  have  been 
tested,  what  problems  were  identified  during  testing,  and  what  corrections 
are  being  made.  Developing  software  without  configuration  management 
frequently  results  in  projects  that  are  delivered  late,  exceed  budget,  and 
perform  poorly. 

Officials  in  doe’s  Office  of  Nonproliferation  and  National  Security  agreed 
that  the  replacement  nmmss  software  had  not  been  placed  under 
configuration  management  at  the  time  of  our  exit  conference.  The  officials 
stated  that,  until  recently,  they  had  not  required  the  use  of  configuration 
management  on  software  development  projects.  In  its  written  comments 
on  a  draft  of  this  report,  doe  stated  that  the  replacement  nmmss  is  now 
being  placed  under  configuration  control. 


DOE  Does  Not  Plan  to 
Adequately  Test  NMMSS 


During  acceptance  testing,  tests  are  performed  to  determine  if  a  system 
will  meet  its  hardware,  software,  performance,  and  user  operational 
requirements.  Acceptance  testing  is  usually  performed  by  the  system 
developer  and  witnessed  by  an  independent  verification  and  validation 
group,  which  includes  system  users.  Such  testing  is  important  to 
determine  if  the  new  system  performs  as  required. 

The  previous  implementation  schedule  for  the  replacement  nmmss  called 
for  acceptance  testing  and  2  months  of  parallel  operation  with  the  existing 
NMMSS.  In  addition,  in  a  January  1994  memorandum,  an  official  from  doe  s 
Office  of  Nonproliferation  and  National  Security  stated  that  the 
replacement  nmmss  would  not  be  made  operational  until  it  has  been 
demonstrated  that  the  new  system  is  capable  of  meeting  present  and  new 
customer  needs  and  requirements.”  Adhering  to  this  position  on  testing  the 
replacement  NMMSS  would  have  greatly  reduced  system  risks. 

In  January  1995,  doe  changed  its  position  and  decided  to  make  the 
replacement  nmmss  operational  without  performing  acceptance  testing. 

DOE  officials  stated  that  this  decision  was  made  to  avoid  the  cost  of 
simultaneously  funding  both  the  existing  and  replacement  systems. 

Instead,  doe  plans  to  perform  what  it  is  calling  “system  testing”  on  a  subset 
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of  system  reports — 87  of  approximately  500  reports.  While  doe  stated  that 
these  87  reports  were  selected  based  on  users’  needs,  it  could  not  produce 
documentation  to  validate  this  statement. 

The  only  system  testing  at  the  time  of  our  review  was  the  informal  testing 
that  the  subcontractor  stated  it  had  performed.  However,  the 
subcontractor  could  not  provide  documentation  on  either  its  test  plans  or 
the  test  results.  In  its  written  comments  on  a  draft  of  this  report,  doe 
officials  stated  that  system  test  procedures  have  now  been  written  and 
approved. 

In  addition,  parallel  operations  with  the  existing  nmmss  are  not  scheduled. 
During  parallel  operations,  both  systems  would  perform  all  required 
functions,  and  then  outputs  would  be  compared  to  ensure  that  the 
replacement  system  is  producing  accurate  reports.  Because  the 
replacement  nmmss  will  replicate  the  functions  of  the  existing  nmmss,  a 
period  of  parallel  operations  is  especially  important. 

Without  parallel  processing,  doe  is  introducing  additional  risk  that  the 
replacement  system  will  not  perform  all  functions  of  the  existing  system 
or,  more  importantly,  that  the  information  produced  by  the  replacement 
system  will  not  be  accurate.  As  a  result,  doe  cannot  guarantee  its  users 
that  the  information  they  need  from  nmmss  to  do  their  jobs  will  continue  to 
be  available,  nmmss  users  told  us  that  information  they  get  from  the 
existing  nmmss  withm  hours  could  take  weeks  or  months  to  gather  if  they 
cannot  obtain  it  from  the  new  nmmss  or  if  they  cannot  be  sure  that  the 
information  in  the  new  nmmss  is  accurate. 


Conclusions 


doe  has  stated  that  it  will  discontinue  the  existing  system  on 
September  1, 1995,  and  begin  operation  of  the  replacement  nmmss  without 
acceptance  testing.  However,  doe’s  replacement  nmmss  is  being  developed 
in  an  imdisciplined,  poorly  controlled  manner  that  makes  success  unlikely. 
Planning  was  inadequate  and  basic  system  development  practices  are  not 
being  followed.  As  a  result,  doe  will  not  know  if  the  replacement  nmmss 
will  produce  the  accurate  and  timely  reports  needed  to  meet  users’  needs 
before  it  accepts  the  system  and  pays  the  subcontractor. 

doe’s  disregard  for  basic  system  development  practices  necessary  to 
ensure  the  accuracy  and  dependability  of  its  nuclear  tracking  system  is 
inconsistent  with  the  importance  of  nmmss,  which  provides  the  United 
States’  official  record  for  tracking  nuclear  materials.  It  is  not  in  doe’s  best 
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Agency  Comments 
and  Our  Evaluation 


interests,  therefore,  to  disconnect  the  existing  nmmss  and  replace  it  with  an 
untested,  undocumented  new  system.  The  history  of  software 
development  is  littered  with  systems  that  failed  under  similar 
circumstances. 


We  recommend  that  the  Secretary  of  Energy  immediately  terminate  any 
further  development  of  the  replacement  nmmss.  Further,  as  we 
recommended  in  our  December  1994  report,  the  Secretary  should  direct 
the  Office  of  Nonproliferation  and  National  Security  to  determine  users’ 
requirements,  investigate  alternatives,  and  conduct  cost-benefit  analyses 
before  proceeding  with  any  plan  to  develop  a  replacement  nmmss. 

If,  after  thorough  planning,  the  Office  proceeds  with  plans  to  develop  a 
new  NMMSS,  we  recommend  that  it  follow  generally  accepted  system 
development  practices.  In  the  interim,  we  recommend  that  doe  continue 
using  the  existing  nmmss  system  imtil  the  above  recommendations  are 
addressed. 


The  Department  of  Energy  provided  written  comments  on  a  draft  of  this 
report.  Their  comments  are  summarized  below  and  reproduced  in 
appendix  I. 

The  Department  of  Energy  agreed  with  the  need  for  systems  development 
documentation,  configuration  management,  and  adequate  testing. 
However,  the  Department  did  not  concur  with  our  assessment  of  its 
analyses  and  planning  for  the  system  development  effort,  or  with  our 
recommendation  that  it  terminate  the  system  development  until  users’ 
requirements,  alternatives,  and  cost-benefit  analyses  have  been  performed. 
DOE  stated  that  its  planning  was  adequate  because  it  is  converting  an 
existing  system  from  an  unstructured  language  to  a  structured,  fourth 
generation  language,  rather  than  developing  a  new  system. 

We  disagree.  Without  sound  analyses  or  planning,  doe  does  not  know  that 
“converting  an  existing  system”  is  a  cost-effective  way  to  meet  its  needs. 
Furthermore,  as  our  report  discusses,  doe  is  implementing  this 
imsupported  approach  in  an  imsatisfactoiy  manner.  Therefore,  doe  should 
discontinue  its  current  effort  and  perform  users’  requirements, 
alternatives,  and  cost-benefit  analyses  before  proceeding. 
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As  arranged  with  your  office,  unless  you  publicly  announce  the  contents  of 
this  report  earlier,  we  plan  no  further  distribution  until  30  days  from  the 
date  of  this  letter.  At  that  time,  we  will  provide  copies  of  this  report  to  the 
Secretary  of  Energy;  the  Director,  Office  of  Management  and  Budget; 
appropriate  congressional  committees;  and  other  interested  parties. 

Copies  will  also  be  made  available  to  others  upon  request. 

Please  call  me  at  (202)  512-6253  if  you  or  your  staff  have  any  questions. 
Major  contributors  to  this  report  are  listed  in  appendix  H. 

Sincerely  yours. 


Joel  C.  Willemssen 
Director,  Information  Resources 
Management/Resources,  Community, 
and  Economic  Development 
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Mr.  Joel  C.  Willems sen 
Director,  Information  Resources 
Management/Resources.  Community 
and  Economic  Development 
General  Accounting  Office 
Washington.  D.C.  20548 

Dear  Mr,  Wi  Hems  sen: 

The  Office  of  Nonproliferation  and  National  Security  appreciates  the 
opportunity  to  review  the  General  Accounting  Office  report,  “DEPARTMENT  OF 
ENERGY:  Poor  Management  of  Nuclear  Materials  Tracking  System  Makes  Success 
I'llikely."  We  have  attached  our  comments  to  this  report  for  your  review.  We 
understand  that  the  General  Accounting  Office  has  recommended  that  the 
Secretary  of  Energy  terminate  any  further  development  of  the  replacement 
'■djclrar  Materials  Management  and  Safeguards  System.  However,  we  are  confident 
that  the  Department's  on-going  actions  will  satisfy  the  General  Accounting 
f.ffice's  concerns  and  that  the  replacement  Nuclear  Materials  Management  and 
Safeguards  System  will  meet  and  eventually  exceed  U.S.  Government 


Sincerely, 

Kenneth  E.  Baker 
Acting  Director 
Office  of  Nonproliferation  and 
National  Security 
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Cofinents  on 
GAO  Draft  Report 
"DEPARTMENT  OF  ENERGY; 

Poor  Management  of  Nuclear  Materials  Tracking  System 
Makes  Success  Unlikely" 


General  Comments: 

The  Department  is  concerned  that  the  General  Accounting  Office  (GAO) 
feels  the  Department  of  Energy  did  not  adequately  plan  the  replacement 
Nuclear  Materials  Management  and  Safeguards  System  (NMMSS) .  We  agree 
that  the  development  of  any  new  system  must  be  responsive  to  users* 
requirements,  include  a  cost-benefit  analysis,  and  undergo  formal 
systems  development.  GAO  correctly  stated  that  the  current  NMMSS  "..is 
housed  on  a  mainframe  using  unstructured  COBOL  code.  Performing 
modifications  on  the  existing  NMMSS  and  designing  custom  reports  are 
(sic)  difficult  because  of  the  volume  and  complexity  of  the  code."  The 
development  of  any  reports  on  the  existing  NMMSS  system  is  laborious  and 
time  consuming.  Therefore,  for  the  Department  to  decide  to  translate 
the  existing  unstructured  software  into  industry  standard  query 
language,  using  modern  PC  technology,  without  adversely  affecting  users' 
requirements  and  at  a  reduced  recurring  cost,  would  appear  to  be  a 
benefit  of  good  planning.  As  stated,  the  Department  is  translating  an 
existing  system  and  not  formally  developing  a  new  system.  Therefore, 
users'  requirements  are  not  an  issue  since  the  current  system  meets 
known  user  needs. 

Specific  Comments; 

DOE  IS  PURSUING  THE  REPLACEMENT  NWSS  WITHOUT  ADDRESSING  CRITICAL 
PLANNING  ISSUES 

The  Department  did  not  concur  with  the  General  Accounting  Office 
criticism  of  the  systems  development  practices  cited  in  GAO  report 
RCED/AIMD-95-5.  The  Department  still  believes  that  its  analysis  and 
review  efforts  were  adequate,  notwithstanding  the  time  and  budget 
constraints  on  its  planning  and  systems  development  efforts.  As  stated 
previously,  the  Department  is  upgrading  its  reporting  capabilities  by 
translating  an  existing  system  and  not  formally  developing  a  new  system. 

LITTLE  SYSTEM  DEVELOPMENT  DOCUMENTATION  EXISTS 

The  Department  agrees  that  there  is  little  formal  system  development 
documentation.  When  the  project  commenced,  emphasis  was  placed  on  the 
translation  of  the  existing  NMMSS  system  rather  than  the  documentation 
of  the  process.  About  halfway  through  the  translation  effort,  an 
advisory  team,  composed  of  computational  experts,  was  formed  to  provide 
systems  development  expertise.  One  of  its  first  initiatives  was  to 
assist  in  the  development  of  product  and  process-related  documentation. 
The  product-related  documents  developed  were  the  system  requirements 
specification  and  the  system  test  plan.  Process-related  documents  were 
the  software  configuration  management  plan,  program  management  plan,  and 
software  quality  assurance  plan.  The  Department  is  aware  that  there  is 
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an  extensive  paper  trail  associated  with  the  software  translation 
effort.  However,  this  paper  trail  Is  not  in  the  form  of  formal  system 
development  documentation.  The  Department  is  rectifying  this  situation. 

CONFIGURATION  MANAGEMENT  WAS  NOT  USED 

The  Department  agrees  that  a  successful  system  project  should  include  a 
software  configuration  management  plan  that  clearly  defines  the 
procedures  for  identifying,  accounting  for,  and  reporting  on  changes  to 
software  items  that  are  under  configuration  control-  NMMSS  software  was 
placed  under  configuration  management  on  April  21,  1995,  and  newly 
completed  modules  have  been  added  since  then.  As  part  of  the  software 
configuration  management  plan,  the  software  is  being  placed  under 
configuration  management  as  modules  are  completed. 

DOE  DOES  NOT  PLAN  TO  ADEQUATELY  TEST  NWSS 

The  Department  is  well  aware  of  the  importance  of  NMMSS  and,  therefore 
plans  to  adequately  test  the  replacement  system.  Since  November,  1994, 
the  new  NMMSS  has  been  processing  the  same  data  that  the  current  NMMSS 
is  processing.  Processing  results  are  provided  to  customers  in  a 
progressive  manner  (select  group  of  customers  in  June,  additional 
customers  in  July,  even  more  customers  in  August).  Extensive  systems 
test  procedures  have  been  written  and  approved.  These  procedures 
include  35  individual  tests,  many  of  which  are  already  being  rehearsed. 
The  complete  systems  test  suite  is  scheduled  to  be  run  in  mid-August. 

The  Department  will  not  allow  the  new  NMMSS  to  become  the  government's 
system  of  record  until  system  testing  is  accomplished. 
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Accounting  and 
Information 
Management  Division, 
Washington,  D.C. 


Valerie  C.  Melvin,  Assistant  Director 
Keith  A.  Rhodes,  Technical  Assistant  Director 
Suzanne  M.  Bums,  Evaluator-in-Charge 
Linda  J.  Lambert,  Senior  Auditor 
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